Highlevel active pen matrix

ABSTRACT

The present invention relates to a system, method and medium for receiving and acting upon user input. In one embodiment, the user may only have access to a limited input device, like a stylus. Using the present invention, a user is provided with intuitive responses from the system based on inputs from the limited input device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 10/993,357, filed Nov. 22, 2004 which is a continuation of U.S.patent application Ser. No. 09/736,170, filed Dec. 15, 2000, now issuedas U.S. Pat. No. 6,897,853, which claims priority to U.S. ProvisionalPatent Application Ser. No. 60/247,400, filed Nov. 10, 2000, each ofwhich is incorporated by reference herein as to its entirety.

FIELD OF THE INVENTION

Aspects of the present invention are directed generally to apparatus andmethods for controlling a graphical user interface (GUI). Moreparticularly, the present invention relates to receiving user input,determining based on the user input what the user wants to do, andperforming a function related to the desired input.

BACKGROUND OF THE INVENTION

Typical computer systems, especially computer systems using graphicaluser interface (GUI) systems such as Microsoft WINDOWS, are optimizedfor accepting user input from one or more discrete input devices such asa keyboard and for entering text, and a pointing device such as a mousewith one or more buttons for driving the user interface. Virtually allsoftware applications designed to run on Microsoft WINDOWS are optimizedto accept user input in the same manner. For instance, many applicationsmake extensive use of the right mouse button (a “right click”) todisplay context-sensitive command menus. The user may generate othergestures using the mouse such as by clicking the left button of themouse (a “left click”), or by clicking the left or right button of themouse and moving the mouse while the button is depressed (either a “leftclick drag” or a “right click drag”).

In some environments, a mouse is not usable or desirable. For example,in a digitizer tablet environment, the primary input device may be astylus. While a stylus attempts to provide pad and paper-like feel to acomputing environment, current systems are limited. For example, the useof a stylus in a graphical user interface is limited to tapping onvarious items for selection. See, for example, the Palm-series ofproducts using the Palm OS 3.0 operating system. Further, instylus-based input environments, a user is continually forced to selecttools or operations from a remote tool bar, generally on a top or bottomof a screen. While a user can type in letters or have the digitizerrecognize handwriting, these operations require selecting a keyboardinput mode and writing in a predefined portion of the digitizer,respectively. In short, requiring a user to tell the computer, for everynew input, what a user wants to do makes stylus-based computingdifficult for the average user. Accordingly, stylus based inputs havebeen relegated to personal data assistants (PDAs) where significant userinput is not possible. Mainstream computing still requires the use of atleast a keyboard and mouse (or mouse-based input device, for example,trackballs, touch-pads, and other mouse substitutes).

Accordingly, a need exists for permitting a user to perform alloperations of a mouse-type device using a stylus.

SUMMARY OF THE INVENTION

As discussed in the various copending patent applications incorporatedherein by reference, aspects of the present invention are directed to atablet-like computer that allows users to directly write on a displaysurface using a stylus. The display surface may physically, optically,and or electro magnetically detect the stylus. The computer may allowthe user to write and to edit, manipulate, and create objects throughthe use of the stylus. Many of the features discussed in these copendingapplications are more easily performed by use of the various aspects ofthe present invention discussed herein.

An aspect of the present invention is directed to methods and apparatusfor simulating gestures of a mouse by use of a stylus on a displaysurface. The present invention determines the operation a user wants toperform based on the user's input. This determination may includereference to other information including the location of the user'sinput on a digitizer (e.g., location on a screen) and the status ofother objects or elements as displayed. By using this information, thesystem determines what the user wants to do and implements the action.

A number of inputs with a stylus are possible. For example, a user maytap a stylus, stroke the stylus, hold the stylus at a given point, orhold then drag the stylus. Other inputs and combinations are possible asnoted by the above-identified applications, which are expresslyincorporated herein by reference.

As to a stroke operation, the system may drag an object, may maintain acurrent state or operation, or being inking. Inking may include writing,drawing, or adding annotations as described in greater detail in U.S.Ser. No. 60/212,825, filed Jun. 21, 2000, entitled “Methods forClassifying, Anchoring, and Transforming Ink Annotations” andincorporated by reference.

As to a tap operation, the system may add to existing writing, mayselect a new object, insert a cursor or insertion point, or may performan action on a selected object.

As to a hold operation, the system may simulate a right mouse buttonclick or other definable event.

As to a hold and drag operation, the system may drag a selected objector perform other functions.

These and other features of the invention will be apparent uponconsideration of the following detailed description of preferredembodiments. Although the invention has been defined using the appendedclaims, these claims are exemplary in that the invention is intended toinclude the elements and steps described herein in any combination orsubcombination. Accordingly, there are any number of alternativecombinations for defining the invention, which incorporate one or moreelements from the specification, including the description, claims, anddrawings, in various combinations or subcombinations. It will beapparent to those skilled in the relevant technology, in light of thepresent specification, that alternate combinations of aspects of theinvention, either alone or in combination with one or more elements orsteps defined herein, may be utilized as modifications or alterations ofthe invention or as part of the invention. It is intended that thewritten description of the invention contained herein covers all suchmodifications and alterations.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary of the invention, as well as the followingdetailed description of preferred embodiments, is better understood whenread in conjunction with the accompanying drawings, which are includedby way of example, and not by way of limitation with regard to theclaimed invention. In the accompanying drawings, elements are labeledwith three-digit reference numbers, wherein the first digit of areference number indicates the drawing number in which the element isfirst illustrated. The same reference number in different drawingsrefers to the same element.

FIG. 1 is a schematic diagram of a general-purpose digital computingenvironment that can be used to implement various aspects of theinvention.

FIG. 2 is a plan view of a tablet computer and stylus that can be usedin accordance with various aspects of the present invention.

FIGS. 3-7 are flowcharts showing a variety of steps for interpreting auser's input in accordance with embodiments of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention may be more readily described with reference toFIGS. 1-7. FIG. 1 illustrates a schematic diagram of a conventionalgeneral-purpose digital computing environment that can be used toimplement various aspects of the present invention. In FIG. 1, acomputer 100 includes a processing unit 110, a system memory 120, and asystem bus 130 that couples various system components including thesystem memory to the processing unit 110. The system bus 130 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory 120 includes read only memory (ROM)140 and random access memory (RAM) 150.

A basic input/output system 160 (BIOS), containing the basic routinesthat help to transfer information between elements within the computer100, such as during start-up, is stored in the ROM 140. The computer 100also includes a hard disk drive 170 for reading from and writing to ahard disk (not shown), a magnetic disk drive 180 for reading from orwriting to a removable magnetic disk 190, and an optical disk drive 191for reading from or writing to a removable optical disk 192 such as a CDROM or other optical media. The hard disk drive 170, magnetic disk drive180, and optical disk drive 191 are connected to the system bus 130 by ahard disk drive interface 192, a magnetic disk drive interface 193, andan optical disk drive interface 194, respectively. The drives and theirassociated computer-readable media provide nonvolatile storage ofcomputer readable instructions, data structures, program modules andother data for the personal computer 100. It will be appreciated bythose skilled in the art that other types of computer readable mediathat can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, Bernoullicartridges, random access memories (RAMs), read only memories (ROMs),and the like, may also be used in the example operating environment.

A number of program modules can be stored on the hard disk drive 170,magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including anoperating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a microphone, joystick, game pad, satellite dish, scanner orthe like. These and other input devices are often connected to theprocessing unit 110 through a serial port interface 106 that is coupledto the system bus, but may be connected by other interfaces, such as aparallel port, game port or a universal serial bus (USB). Further still,these devices may be coupled directly to the system bus 130 via anappropriate interface (not shown). A monitor 107 or other type ofdisplay device is also connected to the system bus 130 via an interface,such as a video adapter 108. In addition to the monitor, personalcomputers typically include other peripheral output devices (not shown),such as speakers and printers. In a preferred embodiment, a pendigitizer 165 and accompanying pen or stylus 166 are provided in orderto digitally capture freehand input. Although a direct connectionbetween the pen digitizer 165 and the processing unit 110 is shown, inpractice, the pen digitizer 165 may be coupled to the processing unit110 via a serial port, parallel port or other interface and the systembus 130 as known in the art. Furthermore, although the digitizer 165 isshown apart from the monitor 107, it is preferred that the usable inputarea of the digitizer 165 be co-extensive with the display area of themonitor 107. Further still, the digitizer 165 may be integrated in themonitor 107, or may exist as a separate device overlaying or otherwiseappended to the monitor 107.

The computer 100 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer109. The remote computer 109 can be a server, a router, a network PC, apeer device or other common network node, and typically includes many orall of the elements described above relative to the computer 100,although only a memory storage device 111 has been illustrated inFIG. 1. The logical connections depicted in FIG. 1 include a local areanetwork (LAN) 112 and a wide area network (WAN) 113. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

When used in a LAN networking environment, the computer 100 is connectedto the local network 112 through a network interface or adapter 114.When used in a WAN networking environment, the personal computer 100typically includes a modem 115 or other means for establishing acommunications over the wide area network 113, such as the Internet. Themodem 115, which may be internal or external, is connected to the systembus 130 via the serial port interface 106. In a networked environment,program modules depicted relative to the personal computer 100, orportions thereof, may be stored in the remote memory storage device.

It will be appreciated that the network connections shown are exemplaryand other techniques for establishing a communications link between thecomputers can be used. The existence of any of various well-knownprotocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed,and the system can be operated in a client-server configuration topermit a user to retrieve web pages from a web-based server. Any ofvarious conventional web browsers can be used to display and manipulatedata on web pages.

FIG. 2 illustrates a tablet PC 201 that can be used in accordance withvarious aspects of the present invention. Any or all of the features,subsystems, and functions in the system of FIG. 1 can be included in thecomputer of FIG. 2. Tablet PC 201 includes a large display surface 202,e.g., a digitizing flat panel display, preferably, a liquid crystaldisplay (LCD) screen, on which a plurality of windows 203 is displayed.Using stylus 204, a user can select, highlight, and write on thedigitizing display area. Examples of suitable digitizing display panelsinclude electromagnetic pen digitizers, such as the Mutoh or Wacom pendigitizers. Other types of pen digitizers, e.g., optical digitizers, mayalso be used. Tablet PC 201 interprets marks made using stylus 204 inorder to manipulate data, enter text, and execute conventional computerapplication tasks such as spreadsheets, word processing programs, andthe like.

A stylus could be equipped with buttons or other features to augment itsselection capabilities. In one embodiment, a stylus could be implementedas a “pencil” or “pen”, in which one end constitutes a writing portionand the other end constitutes an “eraser” end, and which, when movedacross the display, indicates portions of the display are to be erased.Other types of input devices, such as a mouse, trackball, or the likecould be used. Additionally, a user's own finger could be used forselecting or indicating portions of the displayed image on atouch-sensitive or proximity-sensitive display. Consequently, the term“user input device”, as used herein, is intended to have a broaddefinition and encompasses many variations on well-known input devices.

Region 205 shows a feed back region or contact region permitting theuser to determine where the stylus as contacted the digitizer. Inanother embodiment, the region 205 provides visual feedback when thehold status of the present invention has been reached.

FIGS. 3-7 show various flowcharts for determining what a user wants todo based on a user's interaction with the digitizer. As will bediscussed below, the user contacts the digitizer where the user wants tobegin writing, tapping, annotating, dragging, etc. In the case where thedigitizer is superimposed over a display, the user's contact with thedigitizer is directed at operating at (or near) the contact pointbetween the user's stylus and the currently displayed information at ornear the contact point.

In step 301, the system senses a contact or other indication of anaction. In one embodiment the contact may be the stylus contacting thesurface of the digitizer. In another embodiment, the action may bebringing the tip of the stylus near the digitizer's surface. Further, ifthe stylus includes another signaling method (for example, a radiotransmitter transmitting a signal to the digitizer signaling a user'sinput), the digitizer (or related input mechanism or mechanisms)interpret the received signal as a user's input. Other methods ofstarting an operation or writing or contact with a digitizer are knownin the art. For purposes of illustration and description, the system andmethod reference physical contact with the digitizer. All other ways ofproviding signals to a processor are considered within the scope of theinvention and are not mentioned here for simplicity.

In step 302, the system determines the contact position and what liesbeneath the contact position (for example, an object, a drawing, blankspace, ink, and the like). In step 303, the system determines if thestylus has moved beyond a first threshold (time, distance, rate, oracceleration, and the like). In one embodiment, the threshold is set tothe minimum resolvable movement. In another embodiment, the threshold isset higher to account for shaky hands, vibrations of the digitizer ortablet pc (for example, if trying to use the system while driving in acar over a bumpy road). It is noted that objects may have all the samethreshold. Alternatively, objects may have different thresholds. Thismay be dependent on the object, the size of the object, the state of thesystem, the state of the object, and the like.

If the first threshold has been exceeded, then the system proceeds tostep 304 where the user's input is classified as a stroke and the systemsteps to point A 305. If the first threshold has not been exceeded, thesystem determines if the stylus was still in contact with the digitizerwhen a time threshold had expired in step 306. If no (meaning that thestylus was still in contact with the digitizer surface), the systemclassifies the input as a tap in step 307 and proceeds to point B 308.

If the stylus was still in contact with the surface after the timethreshold in step 306, the system determines if a second move thresholdwas exceeded in step 309. The first and second move thresholds may beidentical or different. For example, both may be 0.25 mm. Or, the firstmay be 0.5 mm or one mm and the second be 0.3 mm. Further, the first maybe 1.2 mm or more and the second may be 0.5 mm or more. In short, anyvalues may be used as long as they are not obtrusive to the user. Thesecond threshold may be determined only after the time threshold of step306 has expired. In this example, the second threshold may be higherthan the first threshold (or it may be the same or smaller).

If the second move threshold was not exceeded, then the systemclassifies the input as a hold in step 310 and proceeds to point C 311.If the second move threshold was exceeded, then the system classifiesthe input as a ‘hold and drag’ in step 312 and moves to point D 313.

FIG. 4 shows point A as starting point 401. Here, the system classifiedthe input as a stroke and begins stroke processing in step 402. In step403, the system determines if the stroke started on a draggable object.If yes, the system determines in step 404 whether drag threshold wasexceeded (for example, 0.25 inches, 0.25 inches per second and thelike). If so, the system classifies the stroke as a drag in step 405 andperforms a function that is dependent on the object. For example, thedrag may extend a selection as described in greater detail in “SelectionHandles in Editing Electronic Documents,” filed concurrently with thepresent application Ser. No. 60/247,973, and expressly incorporated byreference. Also, the drag may operate a bungee tool as described in Ser.No. 60/247,842, entitled “Insertion Point Bungee Space Tool”, and filedconcurrently with the present application, and expressly incorporatedherein.

If, in step 404, the drag threshold has not been exceeded, the systemmaintains the current state (with the object being selected or not) instep 407. If the stroke was not over a draggable object in step 403, thesystem determines if the area under the contact point is inkable in step408. For example, inkable may mean an area capable of receiving ink(including drawings, annotations, or writing) as detailed in Ser. No.60/212,825, filed Jun. 21, 2000, and expressly incorporated herein byreference for essential subject matter. By contrast, a control button(for copy, save, open, etc.) may not be inkable. If inkable in step 408,the system permits inking (drawing, writing, annotating and otherrelated functions) in step 409. If not inkable, the system maintains thecurrent state (objects selected or not) in step 407.

In FIG. 5A, the system starts at point B 501 and operates on the inputas a tap 502. The system determines whether the tap was on an area orobject that is inkable in step 503. If yes, the system determineswhether any ink was recently added or “wet” (for example, less than 0.5or 1 second old) in step 504. If so, the system considers the tap as adot to be added to the ink in step 505 (and adds the dot). If no wet inkexists, then the system determines if the tap was over a selectableobject in step 506. It is noted that steps 503 and 504 may be combined.If the tap was over a selectable object, then the system determines ifthe object was already selected in step 507. If it was not, then thesystem selects the tapped object in step 508. If a previous object hadbeen selected, the system cancels the previous or old selection in step509. If the object was previously selected as determined by step 507,the system performs an action relevant to the object in step 510. Thisaction may include editing the object, performing a predefined operation(for example, enlarge, shrink and the like). From step 506, if the tapwas not on a selectable object, then the system proceeds to point BB512.

FIG. 5B shows additional processing to FIG. 5A. As point BB 512, thesystem determines if the tap was in a space between text (referred toherein as an inline space) in step 513. If yes, the system places aninsertion point at the tap point in step 514. As shown in a broken linedbox, the system may also cancel any old or previous selections in step515. If no, then the system determines if the tap point has ink nearbyin step 518. If the system determines that the tap was nearby ink, thenthe system adds a dot to the ink in step 516. If there was an oldselection, then the system cancels the old selection in step 517 (asshown by a broken line box).

If not nearby ink in step 518, the system determines if the tap is on anactive object in step 519. If the tap was not on an active object, thesystem places an insertion point at the tap point or performs some otherdefinable action in step 520. Again, if there was an old selection, thenthe system cancels the old selection in step 521 (as shown by a brokenline box). If the tap was on an active object as determined by step 519,the system performs an action in step 522. The action may be definableby the user or relate to any function desirable. In one embodiment, theaction may be to perform a function to operate a selection handle orbungee space tool as described in Ser. No. 60/247,973, “SelectionHandles in Editing Electronic Documents,” filed concurrently with thepresent application and expressly incorporated by reference. Also, thedrag may operate a bungee tool as described in Ser. No. 60/247,842,entitled “Insertion Point Bungee Space Tool”, and filed concurrentlywith the present application, and expressly incorporated herein. Otheroperations are known in the art and incorporated herein.

FIG. 6 relates to holding a stylus beyond a time threshold. Startingfrom point C 601, the system classifies the user input as a holdoperation in step 602. Next, the system simulates a right mouse buttonclick or other definable event in step 603. The functions associatedwith step 603 are described in greater detail in U.S. application Ser.No. 60/247,844, entitled “Simulating Gestures of a Mouse Using a Stylusand Providing Feedback Thereto”, filed Nov. 10, 2000, whose contents areexpressly incorporated herein by reference.

FIG. 7 relates to holding a stylus beyond a time threshold and movingthe stylus. Starting from point D 701, the system classifies the userinput as a hold and drag operation in step 702. Next, in step 703 thesystem drags the selected object as directed by the user.

There are a number of alternatives associated with dragging. If the holdand drag relates to an inline space, the system may use this hold anddrag function to select text. Similarly, one may use this function toselect a drawing encountered by the dragged stylus. Further, one mayselect both text and drawings in this manner. Also, the cursor's pointmay become a selection tool that leaves a trail behind it. In thisregard, the user may loop a number of objects, drawing or text in thisregard. The looping of the objects may result in the selecting of theobjects.

An alternate embodiment of the present invention relates to modifyingink drawings or annotations. For example, if one added an annotation(from step 409) to text, one may manipulate the text (for example, byinserting new text) and have the annotation track the manipulation ofthe text. So, if one circled text then added text to the circled text,the annotation would expand to include the added text as well. This isdescribed in relation to in U.S. Ser. No. 60/212,825, filed Jun. 21,2000, entitled “Methods for Classifying, Anchoring, and Transforming InkAnnotations” and incorporated by reference.

While exemplary systems and methods embodying the present invention areshown by way of example, it will be understood, of course, that theinvention is not limited to these embodiments. Modifications may be madeby those skilled in the art, particularly in light of the foregoingteachings. For example, each of the elements of the aforementionedembodiments may be utilized alone or in combination with elements of theother embodiments.

1. A computer-readable medium storing computer-executable instructionsfor performing steps comprising: receiving a user input to a digitizer;classifying the user input as a stroke if the user input satisfies afirst move threshold; classifying the user input as a tap if the userinput fails to satisfy both the first move threshold and a timethreshold; classifying the user input as a hold if the user inputsatisfies the time threshold but fails to satisfy both the first movethreshold and a second move threshold different from the first movethreshold, wherein after determining the hold, triggering a definableevent; and classifying the user input as a hold and drag if the userinput satisfies both the time threshold and the second move thresholdbut fails to satisfy the first move threshold.
 2. The computer-readablemedium of claim 1 wherein receiving the user input to the digitizerincludes detecting a user's finger by the digitizer.
 3. Acomputer-readable medium storing computer-executable instructions forperforming steps comprising: receiving user input to a digitizer; firstdetermining whether the user input moves at least a first distance;second determining whether the user input ends before a certain amountof time; and responsive to the user input failing to move at least thefirst distance within the certain amount of time and failing to endbefore the certain amount of time, third determining whether the userinput moves at least a second distance larger than the first distance,wherein if the user input satisfies the first determining step andsecond determining step but fails to satisfy the third determining step,classifying the user input as a tap and triggering a definable event. 4.The computer-readable medium of claim 3, wherein the user input iscaused by a stylus contacting the digitizer, and the user input endswhen the stylus no longer contacts the digitizer.
 5. Thecomputer-readable medium of claim 3, wherein the computer-executablesteps are further for performing steps of: responsive to the user inputmoving at least the first distance as determined by the step of firstdetermining, classifying the user input as a first type of input;responsive to the user input ending before the certain amount of time asdetermined by the step of second determining, classifying the user inputas a second type of input different from the first type of input; andresponsive to an outcome of the step of third determining, classifyingthe user input as either a third type of input different from the firstand second types of input or a fourth type of input different from thefirst, second, and third types of input.
 6. The computer-readable mediumof claim 3, wherein the computer-executable instructions are further forperforming a step of, responsive to the user input moving at least thefirst distance as determined by the step of first determining, fourthdetermining whether the input begins on a draggable object.
 7. Thecomputer-readable medium of claim 6, wherein the computer-executableinstructions are further for performing a step of, responsive to theuser input beginning on a draggable object as determined by the step offourth determining, fifth determining whether the user input satisfies adrag threshold.
 8. The computer-readable medium of claim 3 wherein theuser input is caused by a digitizer detecting a user's finger in contactwith the digitizer, and the user input ends when the digitizer no longerdetects the user's finger in contact with the digitizer.
 9. Acomputer-readable medium storing computer-executable instructions forperforming steps comprising: receiving a user's input; and firstdetermining whether the input is a stroke based on a first movethreshold; if the input is not a stroke, then second determining whetherthe input is a tap based on a time threshold and triggering a definableevent; if the input is neither a stroke nor a tap, then thirddetermining whether the stroke is a hold based on a second movethreshold larger than the first move threshold, wherein said inputsatisfies said time threshold and said input does not satisfy saidsecond move threshold; or a hold and drag when the input satisfies saidtime threshold and said input satisfies said second move threshold. 10.The computer-readable medium of 9, wherein, if said input satisfies saidfirst move threshold, the input is classified as a stroke.
 11. Thecomputer-readable medium of claim 9, wherein, if said input does notsatisfy said first move threshold and said input does not satisfy saidtime threshold, the input is classified as a tap.
 12. Thecomputer-readable medium of claim 9, wherein the computer-executableinstructions are further for performing a step of simulating a rightmouse click responsive to said input being classified as a hold.
 13. Thecomputer-readable medium of claim 9, wherein the computer-executableinstructions are further for performing a step of dragging a selectedobject responsive to said input being classified as a hold and drag. 14.The computer-readable medium of claim 9 wherein the user's input isreceived at a touch-sensitive display that detects contact with a user'sfinger.